Catch trial performance

## [1] "Excluded 11 participants based on catch-trial performance."

Exclusion of random guesses

We further exclude participants who seem to provide random ratings independent of the scene that they are seeing. We quantify this by computing the mean rating for each utterance across all trials for each participant and computing the correlation between a participant’s actual ratings and their mean rating. A high correlation is unexpected and indicates that a participant chose ratings at random. We therefore also exclude the data from participants for whom this correlation is larger than 0.75.

## Warning in cor(rating_m, rating_m_overall): the standard deviation is zero
## [1] "Excluded 1 participants based on random responses."

Aggregated results

Individual responses

AUC computation

Method 1

We use the AUC function with the splines method to directly compute the AUC.

t-test and regression model with control variables:

## 
##  Two Sample t-test
## 
## data:  aucs.cautious$auc_diff and aucs.confident$auc_diff
## t = -0.31651, df = 180, p-value = 0.752
## alternative hypothesis: true difference in means is not equal to 0
## 95 percent confidence interval:
##  -7.263266  5.255243
## sample estimates:
## mean of x mean of y 
##  10.83339  11.83740
## 
## Call:
## lm(formula = auc_diff ~ cond + test_order + first_speaker_type * 
##     test_order + confident_speaker, data = rbind(aucs.cautious, 
##     aucs.confident))
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -44.853 -10.274   0.286  10.833  47.037 
## 
## Coefficients:
##                                                    Estimate Std. Error
## (Intercept)                                          22.171      3.007
## condconfident (might-biased)                          1.004      2.497
## test_orderreverse                                     3.632      3.440
## first_speaker_typeconfidentfirst                    -27.217      3.564
## confident_speakerconfidentm                          -1.328      2.508
## test_orderreverse:first_speaker_typeconfidentfirst    1.483      5.006
##                                                    t value Pr(>|t|)    
## (Intercept)                                          7.373 6.29e-12 ***
## condconfident (might-biased)                         0.402    0.688    
## test_orderreverse                                    1.056    0.293    
## first_speaker_typeconfidentfirst                    -7.637 1.36e-12 ***
## confident_speakerconfidentm                         -0.529    0.597    
## test_orderreverse:first_speaker_typeconfidentfirst   0.296    0.767    
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 16.85 on 176 degrees of freedom
## Multiple R-squared:  0.3943, Adjusted R-squared:  0.3771 
## F-statistic: 22.92 on 5 and 176 DF,  p-value: < 2.2e-16